Method for training a crew member of a, in particular, military vehicle

ABSTRACT

A method and simulator for training a crew member of a vehicle, especially a military vehicle, in the interior of the vehicle. A camera films the vehicle interior and is to be fixed to the head of the crew member. A display device is to be disposed in the field of view of the crew member and is to be fixed to the crew member&#39;s head. Via markers in the vehicle interior, the position of a view is defined, and at the position of the view, a detail of a virtual vehicle external environment is overlaid in the display device as a function of the position and viewing direction of the crew member.

The instant application should be granted the priority dates of Mar. 24,2010, the filing date of the corresponding German patent application 102010 016 113.6, as well as Mar. 23, 2011, the filing date of theInternational patent application PCT/DE2011/075046.

BACKGROUND OF THE INVENTION

The present invention relates to a method for training a crew member ofa vehicle, especially a military vehicle, in the interior of thevehicle, whereby a virtual vehicle external environment is produced, andalso relates to a corresponding vehicle simulator for training a crewmember.

To train a vehicle crew member, it is advantageous that the operation ofthe vehicle, or of a partial system of the vehicle, that is to bepracticed be carried out in an environment that is as true to theoriginal as possible. Of particular advantage is especially if thetraining takes place in the vehicle itself or in a vehicle interior thatsimulates the vehicle. It is furthermore advantageous if the vehicle isnot actually placed into movement, but rather if the vehicle externalenvironment is merely simulated, because in so doing the vehicle, inparticular the drive components, do not have to be a part of the vehiclesimulator. Furthermore, situations that are relevant to training can besimulated via the generation of virtual realities in a planned manner.

One method for training a crew member of a military vehicle is describedin EP 0 947 797 A2, which discloses that the training of the tank crewmembers is carried out in stationary or permanent simulators, wherebythe simulators represent the original tanks. By means of a simulationcomputer, a virtual vehicle external environment is produced, wherebythis external view is illustrated on three image screens or monitors.

The drawback of this method is that with modern vehicles, especiallymilitary vehicles, a plurality of viewing windows are present, forexample front, side or rear windows, periscopes or other optics, throughwhich the crew member can see the vehicle external environment, so thata plurality of monitors is required. The contents of these monitors thatdisplay the virtual external environment must additionally becoordinated with the position and the direction of viewing of thestudent in order to achieve a high degree of near reality.

It is an object of the present invention to reduce the number ofmonitors that display the virtual external environment.

SUMMARY OF THE INVENTION

The method of the present invention realizes the object with thefeatures of providing a camera that is configured to film the vehicleinterior and to be fixed to the head of the crew member; providing adisplay device that is configured to be disposed in a field of view ofthe crew member and to be fixed to the head of the crew member;providing a marker in the vehicle interior; via the marker, defining theposition of a view; and at the position of the view, overlaying in thedisplay device a detail of a virtual vehicle external environment as afunction of a position and viewing direction of the crew member. Aninventive simulator comprises a camera that is configured to film thevehicle interior and to be worn by the crew member fixed to his or herhead; a display device that is configured to be disposed in the field ofview of the crew member and to be worn by the crew member fixed to hisor her head; and a marker that can be disposed in the vehicle interiorand defines the position of a view, whereby by means of a computer, atthe position of the view, a detail of a virtual vehicle externalenvironment can be overlaid in the display device as a function of theposition and viewing direction of the crew member.

The training method of the present invention provides that the crewmember wears a camera that is fixed to the head and films the vehicleinterior, as well as a display device that is fixed to the head and isdisposed in the field of view of the crew member.

Pursuant to the present invention, an in particular planar marker isfurthermore disposed in the interior of the vehicle; by means of thismarker, the position of a view is defined, whereby at the position ofthe view, a detail of the virtual vehicle external environment, as afunction of the position and viewing direction of the crew member, isoverlaid in the display device. Thus, the virtual external environmentno longer has to be displayed by means of one or more display devices,since the virtual external environment, in a superimposed manner, ismerely overlaid at the viewing positions in the display device that isdisposed in the viewing field of the crew member; these positions ofview are defined by the marker or markers in the vehicle interior. Thus,by means of the markers it is possible in the vehicle interior to definea location in which a view should be provided for the student.

Pursuant to one preferred embodiment, the camera image is illustrated onthe display device, and the detail of the virtual vehicle externalenvironment is overlaid in the camera image. Thus, by means of thecamera and the display device, the crew member sees the vehicleinterior. This interposition of the display device should advantageouslyadversely affect the view of the crew member as little as possible, inother words, the crew member should only minimally notice that he or sheis not seeing the vehicle interior directly through his or her own eyes,but rather through the interposition of the camera and the displaydevice. The vehicle interior represents the training space.

Alternatively, the display device can be partially transparent, so thatthe crew member can observe the vehicle interior through the displaydevice, whereby at the position of the view, a detail of the virtualvehicle external environment is overlaid.

The display device and/or the camera can be part of a so-called HeadMounted Display (HMD, “display fixed to the head”), which iscommercially available.

A particularly realistic training situation results if the marker isalso disposed at the position of a real view in the vehicle interior, sothat for the crew member a vehicle utilization is simulated with whichthe merely virtually overlaid view is also at that location at which thecrew member is actually located in the real vehicle. Thus, the positionof a real viewing window, especially a periscope or outer window of avehicle, can be at the position of the view.

In a particularly advantageous manner, the marker is disposed,especially adhesively mounted, on the actual viewing window itself. As aresult, the marker is not visible in the image of the crew member, sinceat this position the virtual image of the external environment isoverlaid.

Pursuant to a particularly preferred embodiment, the viewing directionand/or the relative position of the crew member is determined by meansof the marker, in so doing, for example, the marker cooperates with thecamera in such a way that by means of a special optical trackingprocess, the relative position of the camera to the marker isdetermined, in particular via a computer, from which the direction ofviewing, especially the head direction, and/or the position of the crewmember, results.

Furthermore, pursuant to a particularly preferred embodiment, thedirection of viewing and/or the absolute position of the crew memberand/or and of the head can be determined. For this purpose, an inparticular additional head movement sensor, especially an inertialsensor, can be utilized. If the head movement sensor is employed inaddition to the tracking process, the rapidity and precision of thesystem is increased. In addition, an improved illustration of thevirtual outer world is generated.

For a representation of the vehicle interior on the display device thatis as realistic as possible, the camera should be displayed at the levelof the eyes of the crew member. One camera is preferably used for eacheye of the crew member. The display device can additionally be in aposition to represent a stereoscopic image, thus increasing thecloseness to reality.

If a plurality of views are present in the vehicle, a respective markercan be associated with these positions, so that with regard to eachview, a detail of the virtual vehicle external environment can beillustrated.

The approximation of reality can be further increased by entirelycovering or concealing the viewing field of the crew member. For thispurpose, the display device can be embodied as opaque glasses,especially data glasses.

The marker, or markers, can be configured such that an unequivocalcorrelation results between the marker and the view, as well as anunequivocal positioning between camera and marker. The marker can have acode pattern, for example by being imprinted, whereby this code patternis preferably configured in a checkerboard-like manner. A laminarmarker, for example a paper marker, is preferably used. Known markershaving a plurality of spheres can also be utilized, whereby the positionof the view is established via the position of the spheres.

Individual components of embodiments of the invention are known in thestate of the art and can be utilized with the invention.

For example, a Head Mounted Display is described in U.S. Pat. No.4,884,137. One possibility for combining real and virtual image data isdescribed, for example, in U.S. Pat. No. 6,166,744. A laminar markerhaving a code pattern is described, for example, in DE 10 2004 046 144A1. Tracking methods are described, for example, in DE 2007 060 263 A1or in DE 10 2005 011 616 A1, whereby the latter also discloses a papermarker having a code pattern.

An inventive vehicle simulator for training a crew member of a vehicle,in particular a military vehicle, in the vehicle interior is providedwith a computer for generating a virtual vehicle external environment,with the simulator additionally including a camera that can be worn bythe crew member fixed to the head and that films the vehicle interior,and a display device that can be worn by the crew member fixed to thehead and is disposed in the viewing field of the crew member, as well asfurthermore a marker that can be disposed in the vehicle interior andthat defines a position of a view, whereby by means of the computer, atthe position of the view a detail of the virtual vehicle externalenvironment can be overlaid in the display device as a function of theposition and viewing direction of the crew member. The vehicle simulatoris thus configured in such a way that it can perform the above-describedinventive method.

In other respects, the above-described advantageous further developmentsof the inventive method can also be used in the same manner with theinventive vehicle simulator.

The training in the vehicle can relate to all of the components of thevehicle, in other words, for example to the driving system, weaponsystem and/or communications system, so that for example the driving,firing and/or providing orders can in particular also be practicedindependently of one another.

BRIEF DESCRIPTION OF THE DRAWINGS

One advantageous specific embodiment of the invention will be describedwith the aid of FIGS. 1 to 4, in which:

FIG. 1 shows a combat vehicle as a vehicle simulator;

FIG. 2 shows a Head Mounted Display;

FIG. 3 shows a Head Mounted Display worn by a person;

FIG. 4 is a schematic illustration of one portion of the vehiclesimulator of FIG. 1.

DESCRIPTION OF SPECIFIC EMBODIMENTS

FIG. 1 shows a combat vehicle 1 as a military vehicle 1, in which crewmembers 2 are located in the vehicle interior 16 for training purposes.The combat tank 1 here represents an original combat tank that can beused for training purposes. However, since the vehicle 1 need not bemoved during the training, individual, in particular moving, parts canbe omitted. The vehicle interior 16 should, nonetheless, conform to theoriginal vehicle.

The vehicle 1 is provided with a plurality of views in the form ofperiscopes 3 through which the crew members 3 in the original vehicle 1could observe the actual external environment 18. In the trainingoperation of the vehicle 1, at the location of the viewing windows ofthe views 3, a detail, that is, a section, of a virtual vehicle externalenvironment should be overlayed or superimposed. A simulation computer4, which can be disposed within the vehicle 1 and additionally can becoupled with the vehicle system computer, serves for the generation ofthe virtual external environment so that the actual vehicle operationcan be coupled with the simulation. A Head Mounted Display (HMD) 6, arespective one of which is worn by the crew members 2 that are to betrained, serves for the indication of the virtual vehicle externalenvironment in place of the actual external environment 18.

The virtual vehicle external environment is a representation, generatedby the computer 4, of an imaginary, 3-dimensional environment of thevehicle 1 that includes exclusively computer-generated elements. Adetail of the virtual vehicle external environment is a portion of theoverall virtual vehicle external environment that corresponds to theview of an observer of the virtual vehicle external environment in aspatial direction or through a view 3, that is, a look-out, of thevehicle, in particular a viewing window or a viewing device.

FIGS. 2 and 3 show different embodiments of a Head Mounted Display 6,which is designed as information or data glasses 13, and can be fastenedto the head 11 of the person to be trained 2 by means of a fasteningdevice 12, for example a flexible band. The Head Mounted Display 6 isfurthermore provided with a display device 15, which is comprised of twomonitors, each of which is disposed in front of one of the eyes of thecrew member 2.

The Head Mounted Display 6 is furthermore provided with cameras 14 thatare disposed on the glasses 13 and each one of which is disposed infront of one of the eyes of the crew member 2, thus making possible astereoscopic image representation upon the display devices 15. Thecameras 14 and the display devices 15 are thus fixed to the head, i.e.they follow a change in position or movement of the head 11 of the crewmember 2.

If a crew member 2 wears the Head Mounted Display 6 that is secured tothe head, the camera images of the vehicle interior 16 of the cameras 14are imaged to him or her in real time on the display devices 15, so thathe or she hardly notices the wearing of the data glasses 13, and thusviews the vehicle interior via the interposition of the camera 14.

The Head Mounted Display 6 is furthermore provided with a head sensor 17that is embodied as an inertial sensor and has three degrees of freedomvia which the viewing direction of the head 11 can be determined. Thehead sensor has a compass in order to be able to absolutely measure thedirection of yaw. A headphone 19 can also be provided.

The data connection between display device 15 and computer 4 or camera14 and computer 4, can, as illustrated in FIG. 1, be wireless or, asillustrated in FIGS. 2 and 4, be effected by means of a data cable 8.

The schematic illustration of FIG. 4 shows the training space in thevehicle interior 16 of the vehicle 1 of FIG. 1. The crew member 2 sitsin front of three periscopes, each of which is comprised of a viewingportion 3.1, 3.2, 3.3, an optical unit 10.1, 10.2, 10.3, which is guidedthrough the roof of the vehicle, and a viewing component 9.1, 9.2, 9.3,each of which has a viewing window 5.1, 5.2, 5.3, through which innormal operation, i.e. in non-training operation, the crew member canobserve the vehicle external environment 18.

The training process takes place when the vehicle is not moving. In thesimulation operation, at the location of the viewing windows 5.1, 5.2,5.3, a detail of the virtual external environment simulated by thecomputer 4 is to be represented. For this purpose, laminar paper markersare glued or otherwise stuck to the viewing windows 5.1, 5.2, 5.3; thepaper markers are provided with a unique, checkerboard-like codepattern. Via the camera 14 of the Head Mounted Display 6, and by meansof a tracking process installed in the computer 4, the position of themarkers within the camera image of the vehicle interior 16 can bedetermined and followed, so that at this location, in conformity withthe position and viewing direction of the crew member 2, a detail of thevirtual vehicle external environment produced in the computer 4 can beoverlaid in the camera image. Thus, by means of the camera 14 and thedisplay devices 15, the crew member 2 sees the vehicle interior 16,whereby at the locations of the markers 7.1, 7.2, 7.3 a simulatedexternal environment is overlaid.

The markers 7.1, 7.2, 7.3 thus define the positions of the detailsduring the training of the crew member 2. If the position of the crewmember 2 or the crew member's viewing direction changes, by means of thetracking process via the markers 7.1, 7.2 7.3 and the camera 14, thedetail of the virtual vehicle external environment is correspondinglyfollowed and changed.

During the training process, the crew ember 2 does not see the actualenvironment 18 of the vehicle 1, but rather the virtual environment,which serves as the complete simulation of an external environment. Atthe positions defined by the markers 7.1, 7.2, 7.3, a completelyimaginary environment is thus displayed to the members 2 that are to betrained, whereby this environment can be generated and controlled fortraining purposes.

Thus, pursuant to the present invention, the use of monitors within thevehicle can be eliminated. The number of devices needed to display thevirtual environment is reduced to merely the display device 15 in theHead Mounted Display 6.

A virtual exterior view is played or shown to the crew member 2 via theHead Mounted Display 6, as a consequence of which the crew member cancarry out training, for example even combat exercises, in thisartificial reality. To ensure that the user 2 has the impression oflooking at actually present viewing windows 5.1, 5.2, 5.3 of a periscopeor external window, the image content should be adapted to headmovements. This is aided by a Head Tracking System, which includes theinertial sensor 17 that can be mounted on the head, and by means ofwhich an absolute rotation of the head 11 of the user can be detected.

It is possible for the user 2 to see the actual world through the HeadMounted Display in that the image of the tracking camera 14 that ismounted on the Head Mounted Display 6 is displayed to the user on thedisplay devices 15 of the Head Mounted Display. In so doing, in thisscenario the camera view initially covers or obscures the virtualenvironment. By means of the technique that is also known as stenciling,the camera image that is overlaid over the virtual image can be stampedout at this location and can thus be made transparent. This should occurprecisely at the position at which the optical tracking recognizes amarker 7.1, 7.2, 7.3 in the camera image, whereby offsets in position orrotation can be programmed to that of the marker. The shape of thestamping can be freely defined from the size and shape of the marker7.1, 7.2, 7.3. The spatial position is adapted to the duration oftransmission of that of the marker 7.1, 7.2, 7.3. Since due to theirvarying code patterns the markers 7.1, 7.2, 7.3 are distinguishable forthe system, different stamping shapes can also be utilized for differentmarkers 7.1, 7.2, 7.3.

Due to a movement of the head 11 in front of, for example, a viewingwindow 5.1, 5.2, 5.3 provided with a marker 7.1, 7.2, 7.3, the spatialposition of the virtual viewing window also changes. For example, if theuser 2 comes closer to the window, it becomes larger, and the user seesa greater detail of the virtual external environment. If the user 2observes the window largely from the side, the viewing angle becomesflat, just as the case would be with a real viewing window. The virtualworld that the user 2 observes through this virtual window isrepresented in conformity with the orientation of the head sensor 17.Thus, for example, the virtual horizontal tilts if the head 11 of theuser tilts, whereby turning of the head 11 in the real world alsoresults in a corresponding direction of viewing in the virtual world.

With the marker process, the position of the markers in front of theuser, not however the position of the user 2, in the room is initiallydetected. For this purpose, a system for the absolute recognition ofposition is required. One possibility would be to realize this via themarkers, whereby for this purpose the exact positions and orientationsof the markers 7.1, 7.2, 7.3 relative to one another must be known,which can be achieved by means of a measurement. However, due to the useof the inertial sensor 17 that is fixed to the head, the expenditure ofthe measurement can be saved.

A negligible drawback in the operating mode that was presented is that achange in position of the user manifests itself only upon the positionof the viewing window 5.1, 5.2, 5.3, but not upon the position in thevirtual world. To this extent, the process is preferably usable withsuch training spaces where the person 2 that is to be trained sits andthus does not significantly change his or her position.

By means of an ability to switch to a virtual complete image view in aHead Mounted Display 6, with which exclusively virtual images arerepresented upon the display device 15, it is additionally possible torealize simulation scenarios where the person 2 that is to be trainedfinds himself virtually outside of the vehicle 1, for example “overhatch” or ‘dismounted’, although in reality being disposed within thevehicle 1.

Pursuant to a modification of the exemplary embodiment, the displaydevice 15 can be partially transparent. In this case, it is notnecessary to illustrate the image of the vehicle interior taken by thecamera 14 on the display device 15. Rather, the crew member can perceivethe vehicle interior directly through the partially transparent displaydevice 15. For the representation of the virtual vehicle externalenvironment, with the aid of the camera 14, the positions of the markers7.1, 7.2, 7.3 are determined. At the locations of the markers 7.1, 7.2,7.3, details of the virtual vehicle external environment are overlaid inthe display device 15, whereby the remaining regions are transparent inthe display device 15.

The specification incorporates by reference the disclosure of Germanapplication 10 2010 016 113.6 filed Mar. 24, 2010, as well asInternational application PCT/DE2011/075046 filed Mar. 23, 2011.

The present invention is of course, in no way restricted to the specificdisclosure of the specification and drawings, but also encompasses anymodifications within the scope of the appended claims.

REFERENCE NUMERALS

-   1 Vehicle-   2 Crew Member-   3 Periscope-   4 Computer-   5 Viewing Window-   6 Head Mounted Display-   7 Marker-   8 Data Connector-   9 Viewing Component-   10 Optical Unit-   11 Head-   12 Fastening Device-   13 Data Glasses-   14 Camera-   15 Display Device-   16 Vehicle Interior-   17 Inertial Sensor-   18 Actual Vehicle external Environment-   19 Headphone

The invention claimed is:
 1. A method for training a crew member of avehicle, especially a military vehicle, in the interior of the vehicle,including the steps of: providing a camera that is configured to filmthe vehicle interior and to be fixed to the head of the crew member;providing a display device that is configured to be disposed in a fieldof view of the crew member and to be fixed to the head of the crewmember; providing a marker in the vehicle interior; via said marker,defining the position of a look-out; and at the position of the view,overlaying in said display device a section of a virtual vehicleexternal environment as a function of a position and viewing directionof the crew member.
 2. A method according to claim 1, which includes thefurther steps of illustrating an image of said camera upon said displaydevice, and overlaying the section of the virtual vehicle externalenvironment into the camera image.
 3. A method according to claim 1,wherein said display device is partially transparent.
 4. A methodaccording to claim 1, wherein at least one of said display device andsaid camera are part of a Head Mounted Display.
 5. A method according toclaim 1, wherein said marker is disposed at the position of saidlook-out.
 6. A method according to claim 5, which includes the furtherstep of disposing the position of said look-out at the position of areal viewing window, in particular a periscope or outer window, of thevehicle.
 7. A method according to claim 1, which includes the step ofdisposing, especially adhesively mounting, said marker on a viewingwindow.
 8. A method according to claim 1, which includes the step ofdetermining at least one of the viewing direction and the position ofthe crew member by means of said marker.
 9. A method according to claim8, which includes the step of determining at least one of the viewingdirection and the position of the crew member by means of said markerand said camera via a tracking process.
 10. A method according to claim1, which includes the step of determining at least one of the viewingdirection and the position of the crew member via a head movementsensor, especially an inertial sensor.
 11. A method according to claim1, which includes the step of disposing said camera at a level of theeyes of the crew member.
 12. A method according to claim 11, whichincludes the step of providing one camera for each eye of the crewmember.
 13. A method according to claim 1, wherein said display deviceillustrates a stereoscopic image.
 14. A method according to claim 1,which includes the steps of representing the positions of a plurality ofviews via a respective section of the virtual vehicle externalenvironment, and associating one of said markers with each position. 15.A method according to claim 1, wherein said display device entirelycovers or conceals the viewing field of the crew member.
 16. A methodaccording to claim 1, which includes the step of providing said markerwith a code pattern.
 17. A method according to claim 16, wherein saidcode pattern is checkerboard like.
 18. A method according to claim 1,which includes the step of determining the position of said marker viaan, in particular optical, tracking process.
 19. A vehicle simulator fortraining a crew member of a vehicle, especially a military vehicle, inthe interior of the vehicle, comprising: a computer for generating avirtual vehicle external environment; a camera configured to be wornfixed to the head of the crew member and to film said vehicle interior;a display device configured to be worn fixed to the head of the crewmember and to be disposed in a field of view of the crew member; and amarker configured to be disposed in said vehicle interior, wherein saidmarker defines a position of a look-out, and wherein at the position ofsaid look-out, said computer is configured to overlay, in said displaydevice, a section of the virtual vehicle external environment as afunction of the position and viewing direction of the crew member.